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ABSTRACT: This paper reports the evaluation of a unique three-year program designed to 
support teachers new to a school district. The emphasis of the evaluation process was on the 
participation of the stakeholders in the evaluation process and the development of their abilities 
in evaluation. 

There is educational significance to this study on two levels. First, it is an example of using a 
highly participatory process for the formative evaluation of a teacher induction program. Second, 
the finding that all groups were comfortable with the mentor also being a participant in the 
evaluation of the interns contradicted the conventional wisdom that the roles should be separated. 


Objective 

This paper reports on the evaluation of a unique three-year program designed to support new 
teachers in a school district. It describes an evaluation that sought to develop the evaluation 
sophistication of district leaders and produce a useful evaluation of the district’s teacher 
induction program. In conducting the evaluation, two things were emphasized. 




One was the participation of the stakeholders in the conduct of the evaluation, and the other was 
to integrate qualitative and quantitative data collection and analysis methodologies into the 
evaluation process. The program that was the focus of the evaluation was used to induct teachers 
new to the district, whether they were new to the profession or simply new to the district. The 
teachers union, school district administrators, and the entry year teachers jointly supported the 
program and endorsed this evaluation. 


Theoretical Perspective 

When the Joint Committee on Standards for Educational Evaluation (1988) considered personnel 
evaluation, it made two observations of interest to school district leadership and, hopefully, the 
school community. First, it noted that it in order to educate students effectively, it is necessary 
for the institution to use evaluation to select, retain, and develop quality personnel (p. 5). Second, 
the Committee noted, “However, despite its centrality, educational institutions have often been 
ineffective in carrying out their personnel evaluation responsibilities” (p. 6). Bolman and Deal 
(1997) reflect somewhat the same point when they write about the substantial time, effort, and 
money devoted to evaluating individuals and programs and note that the results of these 
evaluations typically disappear and yet by their existence serve the purpose of giving the 
institutional an image of good management (p. 244). Yet, they also make the point that 
evaluation, whether of individuals or a program, serves multiple purposes. It can at once serve to 
reward and control performance, help people grow, be an opportunity to exercise power, and 
provide an occasion for all to play roles in a shared ritual (p. 266). 

In the current school reform, high stakes testing environment, the task before the district 
administrator and the evaluator is to overcome the ineffectiveness of evaluation - individual and 
program - and actually cause improvement. This means learning or adaptation must take place; 
the individuals being evaluated must learn and change behaviors and the institution must learn 
and change institutional culture. To bring this about it is necessary for evaluators to “recognize 
the arena in which evaluation operates, understand organizational dynamics, tailor their 
procedures and techniques to market the evaluation process effectively, and become one of the 
voices in the organization” (Wholey, Hatry, & Newcomer, 1994, p. 536). Further, Preskill and 
Torres (2000) suggest that such individual and institutional (team) learning occurs when 
individuals participate in a collaborative evaluation process that is guided by dialogue and 
reflection (pp. 26 & 35). Heifetz (1994) seems to concur when he says. “The inclusion of 
competing value perspectives may be essential to adaptive success” (p. 23). Bridges and Groves 
(1999) in winding up their report on politics in personnel evaluation observed that their analysis 
suggested “that viewing personnel evaluation simply in rational, technical terms conceals more 
than it illuminates” (p. 335). 

With this context in mind, the role of the evaluator must go beyond technical assistance. 

Working with the institutional stakeholders in a clinical manner, the evaluator and the 
stakeholders together evolve a highly customized evaluation over time (Preskill & Torres, 2000, 
p. 29). Since, and perhaps in part due to, the Qualitative-Quantitative Paradigm Wars of 1980- 
1995, a greater openness to mixed methods in evaluation has occurred (Caracelli, 2000, pp. 100- 
101). This openness to a variety of methods should facilitate bringing educators of all roles - 
administrative and classroom - together to approach evaluation as a means to promote dialogue, 


to enhance and make more professional satisfying the work of both teachers and administrators 
(Strange & Tucker, 1999, p. 356). This would make all the more important the conceptualization 
of the role of today’s evaluators by the application of “such terms as facilitator, problem solver, 
educator, coach and critical friend” (Caracelli, 2000, p. 103). 

In their research on implementing a new teacher evaluation system, Davis, Pool, and Mits-Cash 
(2000) also found: 

Given what is known about change in educational organizations, it is clear that for a new teacher 
assessment system to continue to affect the lives of students and improve teaching and learning, 
continual up-dates, modifications, and revisions are in order. Especially as the systems move 
toward full implementation, it is imperative that district policy makers have as much feedback 
from the field as possible. This feedback should be frequent, accurate, and derived from a variety 
of data collection methods [italics added], (p. 304) 

Thus, we accepted as a starting point that for the evaluation to be successful the participants had 
to be involved in its process. The stakeholders had to understand and value the information that 
could be derived from the evaluation, and they needed to be willing to become collaborative 
partners in the undertaking. To accomplish that we took as a framework for this project the 
Personnel Involvement Evaluation Model (Newman, Deitchman, & Williams, 1978). This model 
is a four-step process that requires engagement with key stakeholders in all stages. The four 
stages are: 

1. Identify a sample of key stakeholders. 

2. Collaboratively develop the evaluation criteria/instrument. 

3. Member check for validity of criteria/instrument. 

4. Collaboratively analyze/interpret the data. 

This study began with conversations between the district administrator who was responsible for 
the teacher mentoring-evaluation plan and the researcher. Those discussions and discussions with 
the District Review Board, which is responsible for supervising the implementation of the 
program, brought out the presence of the components of a well-designed induction program. 
Dagenais (1996) specifies the five characteristics of successful induction programs to be: (a) they 
provide a clear scope for the program, (b) they identify mentor selection and matching criteria, 

(c) they provide intrinsic and, or, extrinsic rewards for attracting and retaining mentors, (d) they 
provide mentor training, and (e) they evaluate the success of the program. The district’s 
induction program had the first four characteristics clearly present. The fifth characteristic, 
evaluation of the program, was considered insufficient to generate program improvement and 
thus the researchers were approached to undertake this study. 

Description of Program 

The object of this evaluation was the a medium sized (8,000 students), mid-west, suburban 
school district’s Teacher Evaluation Program (TEP) as it was implemented for beginning 
teachers, experienced teachers new to the district, and reassigned teachers as well as educational 
personnel such as librarians and counselors working under the educational personnel certificate 
for the first time. The responsibility for implementation and administration of the TEP was been 
assigned to a District Review Board (DRB). 


The people who are the intended focus of the TEP are referred to as interns in the program’s 
documentation and this evaluation. A second role in the TEP is that of consulting teacher. 
Consulting teachers are experienced teachers assigned by the district to orient, help develop, and 
evaluate interns during their first three years of employment with the district. The consulting 
teachers are to provide feedback for development and to evaluate the interns as part of the re¬ 
employment decision-making process. For each intern there is a designated administrator who 
works with the consulting teacher by discussing the intern’s development plan with the 
consulting teacher and by advising the consulting teacher of concerns related to the intern. The 
designated administrator deals directly with the intern on non-instructional matters such as 
attendance and district personnel procedures and indirectly through the consulting teacher on 
instruction related matters for the first year of intern participation. In the second and third years 
of participation the designated administrator and consulting teacher have equal responsibility for 
evaluation of instructional performance. 

There are at least nine forms specified for use by the consulting teachers and administrators to 
provide official communications with the intern and the DRB. A consulting teacher is directed to 
spend between 20 and 36 hours with his or her intern in classroom observations and conferences 
during the intern’s first year and 13 to 24 hours in the second and third years. A very specific 
timeline of activities and reports is given with specific actions to be carried out in September and 
October, November, between December 1 and February, in February, March, and May. 

Stipends are provided to the consulting teachers to prepare program documents and meet with 
interns, administrators, and the DRB. Class coverage or release time is provided for the 
consulting teacher to do classroom observations and conferences. 


Evaluation Methods 

As noted above, the evaluation began with conversations between the district administrator who 
was responsible for the teacher mentoring-evaluation plan, the DRB, and the researcher. The 
evaluator developed key evaluation concepts from these conversations and used those concepts 
to stimulate discussions with a six person focus group composed of representatives from all 
stakeholder groups in the program (DRB, mentors/consulting teachers, and teachers who had 
recently been mentees/new hires). One of the evaluators took notes summarizing the essence of 
the discussions, checking their accuracy by paraphrasing them to the focus group members until 
the members agreed that the researcher had captured their views. The data was compiled and 
categorized by the evaluators, and the categories were checked with the district administrator. 

There were four anticipated results from this stepped process. First, the evaluation designers, that 
is, both the outside evaluators and the district personnel, identified the issues that were 
meaningful and pertinent for the district at this time. Secondly, the evaluators learned about the 
issues from knowledgeable and motivated stakeholders. Thirdly, the stakeholders became 
sensitive to each other’s views about the program and its components. Finally, the evaluators 
were able to gain consensus concerning the constructs and evaluation questions about data 
needed to be collected. The five essential evaluation questions were: 


1. What is effective in this program as it is currently designed? 

2. What is not effective in this program as it is currently designed? 

3. Considering what is both effective and not effective, what can be improved in this program? 

4. Considering what is both effective and not effective, what should be improved in this 
program? 

5. How do perceptions of the various stakeholders match up with each other? 

Based upon the questions, the conversations, with the district administrator, the evaluators’ 
notes, and the categorized data from the focus group, the evaluators constructed five parallel 
questionnaires - one for each role group in the program. The initial survey was presented to the 
initial focus group for review and reaction. A panel of experts then reviewed these 
questionnaires, and their feedback resulted in a revision that was submitted to a second panel for 
further critiquing. 

After a second revision, the questionnaires were sent to all participants, approximately 200 
people, for anonymous response. The five separate questionnaires were color coded according to 
the target population intended for the survey - consulting teacher/mentor, mentee/newly hired 
teacher, DRB member, field administrator, and central office administrator. The questionnaires 
were sent with a cover letter encouraging their completion and anonymous return to the district 
administrator. As the completed questionnaires came in, the district administrator had them 
bundled for delivery to the evaluators. Approximately one-half of the questionnaires were 
returned. 

While the responses were being entered into a database, the evaluators reviewed and reconfirmed 
with the initiating district administrator, that the questionnaires were providing the type of 
feedback they had hoped for. The quantitative responses were analyzed through descriptive 
statistics, correlations, and analyses of variance. The qualitative (open-ended) responses were 
analyzed by initial coding, focused coding, categorization, and having at least two and 
sometimes three experts in the content area interpret the results. 

The combination of these two approaches is sometimes referred to as mixed methods (Newman 
& Benz, 1998; Newman, Ridenour, Newman, & DiMarco, 2003). The mixed methods approach, 
which was applied in this evaluation, generally starts off with a qualitative inquiry that leads to a 
quantitative assessment, and sometimes returns to a qualitative interpretation. It is highly 
heuristic in nature. In this study we started with the qualitative question, “What should be 
evaluated?” which was followed by a quantitative assessment of the responses to the 
questionnaires. These results along with the analysis of the open-ended questions resulted in a 
qualitative-quantitative analysis of the mentoring-evaluation program. 

In keeping with the Personnel Involvement Evaluation Model, the evaluators engaged the district 
administrator to help them interpret the meaning of and to look for “patterns and surprises” 
(Landwehr & Watkins, 1986, preface) in the raw data as it was displayed in frequency counts, 
means, and correlation tables. This allowed the administrator to become more reflective about 
the questions he was interested in and these questions became more focused than the previously 
asked questions. After identifying the statistically significant items in the data analyses, the 
administrator identified 23 specific questions that were to be explored in more detail. These 


questions are provided below along with the responses made by the evaluators. 

The recursive nature of this process, that is, looking at the outcome of the analyses and noting 
the patterns of expected and unexpected results, allowed the administrator and the evaluators to 
become more reflective. This is similar to Patton’s (1990) concept of theoretical sampling. The 
administrator had so much information - approximately 100 pages of printouts - that he could 
not necessarily answer the original questions. His knowledge of the participants, however, 
allowed him to see beyond the data, inferring depth and meaning from it, and allowed the 
evaluators to understand the meaning of the data in ways that could not have existed without his 
participation and that of other members of the focus group. 

Data Sources 

The data sources were the participants separated by roles: interns (newly hired teachers), 
consulting teachers (mentors), and administrators/managers (building administration, central 
office administrations, and members of the board managing the program). 

Findings 


In analyzing the data, the three management groups (field administrators, central officer 
administrators, and members of the DRB) were grouped together because their numbers were so 
few separately that any analytical results would be too highly questionable. 

The direct responses to the 23 specific questions poised during the review of the raw data are 
provided here in summary form, that is, the statistics, charts, and all but one table upon which 
they are based are not provided herein. The extended responses and underlying statistics, charts, 
and tables can be obtained from the lead author who may be contacted at ltrenta@uakron.edu. As 
initially poised by the stakeholders, five more general questions began the set of questions and 
more detailed ones followed. In responding to the questions, the evaluators reversed the order, 
holding those more general questions to the last. 

1. Which professional development area was most needed? The consulting teachers rated the 
professional development needs of interns more strongly in all four areas S planning, delivery, 
discipline and management, and instructional evaluation S than did the interns and the 
administrators. However, the only difference in opinion that was statistically significant was that 
between the consulting teachers and the administrators concerning the need for professional 
development with regard to discipline and management. The consulting teachers rated the 
interns’ need in this area significantly stronger than did the administrators. 

2. Which [professional development] need most needs to be met? In general, when asked to rate 
how well professional development needs were met, the interns gave the highest ratings and the 
administrators gave the lowest ratings in all areas. There were statistically significant differences 
between the interns and the administrators in the ratings of the interns’ professional development 
needs being met in Planning and Delivery with the interns giving significantly higher ratings in 
both areas. According to the interns and the administrators the professional development need 
that most needs to be met is Planning while the consulting teachers cited Delivery 


3. Are consulting teachers regarded as exemplary teachers? There are no statistically significant 
differences between the groups in their responses. All three groups regarded the consulting 
teachers as exemplary teachers. 

4. Were there any differences between respondent groups in the rankings of instructional 
components? The interns ranked feedback on Planning significantly lower in importance than did 
the consulting teachers and the administrators. The administrators ranked feedback on Discipline 
and Management significantly lower than did the interns and the consulting teachers. 


Table: Rankings of Importance of Feedback 
for Instructional Components 


CTs 

Interns 

Admin. 

Most Important 

Planning" 

Discipline & 
Managements 

Planning" 


Discipline & 
Managements 

Delivery 

Delivery 


Delivery 

Planning" 

Discipline & 
Managements 

Least Important 

Instructional 

Evaluation 

Instructional 

Evaluation 

Instructional 

Evaluation 


■ Significant difference between interns and both consulting teachers and administrators. 

S Significant difference between administrators and both consulting teachers and interns. 

5. Do the consulting teachers and interns differ in their responses to sufficiency and 
appropriateness of teaching function issues? The interns were significantly less certain of what is 
expected of them in the Teacher Evaluation Program (TEP) than were the consulting teachers 
and the administrators of their roles. There were also significant differences in the perceptions 
about the sufficiency of time in that the consulting teachers expressed a significantly stronger 
disapproval of the statement that there is too much time provided for mentoring than do the 
interns. The administrators agreed with the consulting teachers and significantly differed with the 
interns about the amount of time. 

In the examination of issues related to time there are the next four questions to consider: 

6. Did the interns feel there was enough time for the consulting teachers to observe and 
conference? The consulting teachers did not judge their released time for observing and 
conferencing as sufficient. In this they differed with both the interns and the administrators to a 
statistically significant degree. 

7. Do the interns feel they have enough time to meet the requirements of the TEP? 

8. Do the consulting teachers feel they have enough time to meet the requirements of the TEP? 

9. Do the building administrators feel they have enough time to meet the requirements of the 
TEP? 

The consulting teachers, interns, and administrators did judge the time devoted to the TEP as 
sufficient. At the same time, all three groups also agreed, without significant difference, that the 
TEP required a greater time commitment than they had thought it would. 

In terms of the time provided for mentoring, all disagreed with the statement that too much time 














was provided, and the interns and administrators agreed with the statement that not enough time 
was provided. The responses of the interns was significantly more negative than those of both the 
consulting teachers and the administrators both as to too much time being provided and the need 
for more time for mentoring. 

10. How well did the consulting teachers meet the needs of their interns? Are any areas unique? 
The answer appears to be “quite well.” All three groups agree with that. Although the ratings of 
the administrators are lower than that of the interns and consulting teachers in five instances, 
they are still on the positive side in every instance. Dealing with judgments about the quality of 
the feedback, there are no significant differences between the judgments of the three groups in 
this area. Overall the feedback between consulting teachers and interns is judged to be excellent 
for the most part, and it meets the needs of the interns at least 80% of the time. 

11. Is there a correlation between the size of the experience difference between the interns and 
the consulting teachers and their ability to positively interact with each other? The anonymity of 
the questionnaires prevented making this comparison. 

12. Entering into the specific area of Program Design Issues, the first question is How effective 
is the information provided to interns regarding the TEP? The consulting teachers gave the 
highest ratings of the three groups to the four survey statements related to the TEP program 
design. From their perspective, from strongest to weakest ratings, they understand what is 
expected of them, find the handbook helpful, believe the process is clearly explained, and receive 
information on how to prepare for the TEP. The administrators’ ratings track those of the 
consulting teachers but at non-significantly lower levels. The interns also gave positive ratings to 
the four items, but all their responses were significantly lower than those both of the consulting 
teachers and of the administrators. Their highest rating was for understanding what was expected 
of them, followed by the process being clearly explained and the TEP Evaluation Handbook as 
helpful. Their lowest rating went to clear information being given on how to prepare for the TEP. 
Insofar as the interns are concerned the information provided to them is effective, but they rate it 
at a significantly lower level of confidence than either the consulting teachers or the 
administrators. 

13. Does the feedback provided to the interns meet their needs? The interns rated the feedback 
they received as excellent 78.89% of the time and inadequate only 14.87% of the time. All 
groups agreed that the feedback to interns was sufficient to meet their needs. 

14. Does the feedback provided to the consulting teachers by the District Review Board (DRB) 
meet their needs? The consulting teachers rated the feedback from the DRB as excellent 43.1% 
of the time and inadequate 25.5% of the time. Across the board, the DRB feedback provided to 
the consulting teachers was judged excellent less often and inadequate more often than any other 
feedback. 

15. Do the consulting teachers and the administrators agree in their opinions of the interns? The 
consulting teachers judged that their perceptions agreed with those of the administrators 88.49% 
of the time while the administrators said their perception of the interns agreed with that of the 
consulting teachers 83.1% of the time. The difference between the two groups was not 


significant. 


16. How comfortable are all groups in the combined mentor-evaluator role that consulting 
teachers are fulfilling? The differences in the responses of the three groups were not statistically 
significant. They are more than slightly comfortable with the consulting teachers roles as mentor, 
evaluator, and in the combined role. While the difference is not significant, it is interesting to 
note that it was the administrators who expressed the least comfort in the consulting teachers in 
each of the roles and the consulting teachers who expressed the greatest comfort. All three 
groups also express the greatest comfort with the consulting teachers in the role of mentor, 
followed by the combination of roles with the least level of comfort expressed for the consulting 
teachers in the role of evaluator. 

17. How satisfied were the interns regarding the professional development activities provided for 
them? The interns expressed slight to moderate satisfaction with the professional development 
opportunities afforded to them. 

18. How satisfied were the consulting teachers regarding the professional development activities 
provided for them? Consulting teachers more than slightly agree that their professional 
development opportunities have been sufficient and appropriate. Yet, they do believe that they 
would benefit from training other than Pathwise training and that they need additional training to 
be more effective with new teachers. Of the possibilities explicitly addressed, the weakest 
positive agreements for this aspect came in response to statements that the coaching training 
skills was sufficient, that the consulting teachers share strategies, and that the Pathwise training 
was sufficient. The interns gave a significantly lower, yet still positive, response to the statement 
that there is a need for additional training for consulting teachers to work effectively with new 
teachers. Administrators rated the statement comparably to the consulting teachers. 

Moving to the five general questions that led off the list of questions from the administration, we 
have these questions. 

19. Were there significant differences of perceptions among surveyed groups? Yes, there were. 
However, the significant differences were in the strength of the mean responses to particular 
questions or statements and not different in the sense of being opposite responses except in one 
case. In that one case the interns slightly disagreed with the statement that there was not enough 
time provided for mentoring while the consulting teachers and administrators agreed with the 
statement. Other than that one situation, the differences were between slight agreement and 
moderate to strong agreement with the statements rather then one group agreeing and another 
group disagreeing. 

20. Were there significant differences within groups by assignment, number of years of 
experience, or teaching level? Among the interns, there were no significant correlations based on 
years of experience. There were a number of significant correlations between the teaching level 
of the intern and the responses to some of the questions/statements. In general, the significant 
correlations with the elementary level were positive and those with the middle and high school 
levels were negative (that is, the responses tended to be lower than the responses given by those 
not in the that group). From the middle and high school level responses that had a significant 


correlation, one might generalize that the interns in those levels were more likely than the 
elementary interns to perceived that consulting teachers needed to collaborate more and receive 
more training for working with new teachers. However, it must be emphasized again, that the 
actual responses from these groups tended to be positive or supportive of the program as it was 
put into operation. The middle and high school level interns simply were not as positive in their 
expressed perceptions as were the elementary level interns. 

Among the consulting teachers, there were no significant correlations based on years of 
experience. There were several significant correlations between the teaching level of the 
consulting teacher and the responses to some of the questions/statements. Among the consulting 
teachers, most of the statistically significant correlations between level and responses to 
questionnaire prompts dealt with feedback, either that received by the consulting teachers or their 
perceptions of the importance of feedback related to one of the four teaching functions identified 
in the TEP. The one correlation that did not relate to feedback was that the elementary level 
consulting teachers had a negative correlation with the statement that the CTs share strategies to 
facilitate interns. Between the middle and high school level interns and the elementary level 
consulting teachers, each of the three levels had a group giving lower rating to the success of the 
consulting teachers’ sharing of strategies. 

21. What information do we have regarding the mentoring-evaluating issue in our present 
format? and 

22. If there were not differences, then the similarities of perceptions should be noted. Probably 
the most telling data related to these questions were the responses of the three groups about their 
comfort with the consulting teachers role as evaluator and the combined role of mentor and 
evaluator. All groups expressed slight to moderate comfort with the roles. The consulting 
teachers gave the highest rating and the administrators the lowest with the interns in-between. 
None signaled discomfort with the roles and the comfort level was well above the lowest comfort 
level for all three groups. All groups were basically comfortable with the combined roles. 

23. How do the comments support the survey responses? The responses to the open-ended 
questions provided support for the survey findings and some hints on ways to improve the 
mentoring-evaluating situation. The recommendations for improving the mentoring and 
evaluating components of the program can be summed up as more time, more specifically 
scheduled time for all stakeholders, more opportunities to interact, less paperwork, greater 
flexibility in dealing with newly hired teachers who are experienced and have demonstrated their 
competence elsewhere. 


Results and Conclusions of the Evaluation 

The results from all stakeholders supported the induction program. Some stakeholders were more 
positive about some aspects of the programs than others, but everyone believed the program 
worked. At the same time the perceptions of almost all aspects of the program indicated room for 
improvement. Whether the scale was Strongly Agree to Strongly Disagree, Comfortable to 
Uncomfortable, Sufficient or Appropriate to Insufficient or Inappropriate, or Met or Unmet 
(professional development needs), the mean responses rarely were close to the strongest or most 
positive response. The identification of differences between groups led to discussions between 


the stakeholders and a better understanding of the difference in perceptions. 

One interesting result was the comfort levels of all stakeholders in having the mentors serve in 
dual roles, as a mentor and as an evaluator of his or her intern (new teacher). Mentors’ 
successfully serving as evaluators is contrary to most of the current research about mentoring. In 
this case of added note is that it was the administrators who were least comfortable with 
consulting teachers being both mentors and evaluators, although their comfort level was not 
significantly different than either the interns or the consulting teachers. 

Another issue that became evident was the lack of flexibility caused by requiring the application 
of the TEP process to all new teachers in the same way, that is, by not allowing for difference in 
treatment between a newly hired teacher who is a first year teacher and a newly hired teacher 
who was a successful, experienced teacher in another district. 

Two anticipated results were that time is an issue, as is the amount of record keeping. There were 
three interrelated issues raised about time: (a) the need to prioritize time for the mentoring 
process, (b) the need to provide more time for the consulting teachers to devote to the program, 
and (c) the need for consulting teachers and administrators to prioritize time to deal with the 
evaluation requirements. 

The record keeping paperwork was considered too time consuming and not relevant to 
improving performance. 


Use of Evaluation Results 

Again, in line with the Personnel Involvement Evaluation Model, the final report prepared by the 
evaluators was delivered to and discussed with the district administrator. The executive 
summary, which included detailed responses to the 23 specific questions as well as the original 
five focusing questions, was provided to the DRB. The lead evaluators met with the DRB and 
discussed the findings and recommendations with them. The members recognized the overall 
positive nature of the responses from all participant groups. They also noted that the changes for 
improvement of the program illuminated by the results of the evaluation were subtle rather than 
substantial. The DRB intended to use the executive summary both to stimulate conversations 
with the larger stakeholder group and to report the state of the TEP program to their board of 
education. 


Limitations of the Evaluation 

The most significant limitation was that the evaluation of program effectiveness was tied to the 
perceptions of the stakeholders rather than to measures of behavior, due to a short time frame for 
the evaluation and limited funds. The effort to overcome this limitation included aggregating the 
scores (rankings and ratings) and providing for reliability and validity checks using a focus 
group, expert judges, and piloting the instruments. 

Another limitation is the probable presence of aspects of the Hawthorne Effect. That is, some 
participants can be expected to respond to the surveys in what they perceived to be the socially 
desirable manner rather than expressing their own judgments. On the other hand, people who 
have hostile feelings toward the district or the district management can be expected to respond in 


ways that would be perceived as denigrating the program. They express their anger rather than 
their judgment. The same efforts used to overcome the use of stakeholder perceptions rather than 
hard measures of behavior are useful in overcoming the desire to give the socially acceptable 
response and the desire to express anger or inflict harm rather than to express considered 
judgment. 

To give fullest assurance of anonymity, the survey forms did not have a place for name or 
signature and the forms and return envelopes were not coded in any way. This meant that it was 
not possible to directly encourage those who did not return the survey to fill it out. Also, it was 
not possible to solicit opinions from those who did not return the survey by another means, such 
as a telephone survey. As a result, the evaluation is based on the perceptions of those who chose 
to respond to the survey and does not consider the opinions of those who chose not to respond. 

Due to the small number of administrators in the district, in analyzing the data it was necessary 
to combine the administrator groups (central office and building) in order to have a sufficient 
number for the statistical analysis to have a reasonable degree of certainty. 

Significance 

There is educational significance to this study on two levels. First, it is an example, a case study, 
of using a highly participatory process for the formative evaluation of a teacher induction 
program. The education of the evaluators concerning the program enabled them to more 
effectively meet the needs of the stakeholders/client. The step-by-step interaction between the 
evaluators and the client administrator and the stakeholder focus group enabled stakeholders’ 
questions about the program to become more targeted and more insightful. Mixing qualitative 
data collection and analysis with quantitative methods both longitudinally in the course of the 
evaluation and latitudinally in the questionnaires provided a final product that was highly 
meaningful and useful to the stakeholders. Particularly encouraging was that the meanings or 
concepts held by individuals appeared to broaden as the stakeholders listened to and embraced 
the meanings of others in the course of this evaluative process. The evaluators noted an increased 
sophistication in evaluative questions asked and interpretations given at the end of the process as 
compared to the beginning. 

Second, the finding that all groups were comfortable with the mentor also being a participant in 
the evaluation of the interns contradicted the conventional wisdom that the roles should be 
separated. Particularly interesting is that the two groups expressing the highest level of comfort 
with the mentor-evaluator combined role were those most intimately involved in it, the interns 
and the consulting teachers. 
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